Laparoscopic ultrasound robotic surgical system

ABSTRACT

A LUS robotic surgical system is trainable by a surgeon to automatically move a LUS probe in a desired fashion upon command so that the surgeon does not have to do so manually during a minimally invasive surgical procedure. A sequence of 2D ultrasound image slices captured by the LUS probe according to stored instructions are processable into a 3D ultrasound computer model of an anatomic structure, which may be displayed as a 3D or 2D overlay to a camera view or in a PIP as selected by the surgeon or programmed to assist the surgeon in inspecting an anatomic structure for abnormalities. Virtual fixtures are definable so as to assist the surgeon in accurately guiding a tool to a target on the displayed ultrasound image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No. 11/447,668(filed Jun. 6, 2006), which claims priority to U.S. provisionalApplication No. 60/688,013 (filed Jun. 6, 2005), each of which isincorporated herein by reference.

GOVERNMENT RIGHTS STATEMENT

This invention was made with Government support under Grant No. 1 R41RR019159-01 awarded by the National Institutes of Health. The Governmenthas certain rights to the invention.

FIELD OF THE INVENTION

The present invention generally relates to robotic surgical systems andin particular, to a laparoscopic ultrasound robotic surgical systemuseful for performing minimally invasive surgical procedures.

BACKGROUND OF THE INVENTION

Minimally invasive surgery offers many benefits over traditional opensurgery techniques, including less pain, shorter hospital stays, quickerreturn to normal activities, minimal scarring, reduced recovery time,and less injury to tissue. Consequently, demand for minimally invasivesurgery using robotic surgical systems is strong and growing.

Laparoscopy is a type of minimally invasive surgery in which a smallincision is made in the abdominal wall through which an instrumentcalled a laparoscope is inserted to permit anatomic structures withinthe abdomen and pelvis to be seen. The abdominal cavity is commonlydistended and made visible by the instillation of absorbable gas such ascarbon dioxide. Tubes may be pushed through the same or differentincisions in the skin so that probes or other instruments can beintroduced to a surgical site. In this way, a number of surgicalprocedures can be performed without the need for a large or open cavitysurgical incision.

One disadvantage of laparoscopy, however, is the inability to manuallypalpate hidden or solid organs. Laparascopic Ultrasound (“LUS”) allowsthe surgeon to overcome this limitation by providing visualization ofdeeper structures. In fact, even when open cavity operations areperformed, intraoperative ultrasonography may be significantly moresensitive at detecting otherwise occult lesions within anatomicstructures than bimanual palpation.

As an example, intraoperative ultrasonography of the liver is useful ina variety of clinical settings during laparoscopic surgery. Theseinclude: staging and assessment of the liver, includingultrasound-guided needle biopsy, liver tumor ablation, and evaluation ofthe liver prior to laparoscopic liver resection.

For resection procedures, surgeons should have the ability to performaccurate staging of the liver and other sites to rule out metastaticdisease prior to resection. The addition of LUS to standard laparoscopyimproves the diagnosis of metastases over conventional preoperativediagnostic methods.

Ultrasound-directed liver biopsy is an important component of hepaticstaging and assessment. When a lesion is identified by ultrasound,needle biopsy is necessary to confirm the findings histologically.Current practice requires manual free-hand LUS in conjunction withfree-hand positioning of the biopsy needle under ultrasound guidance.

For the treatment of unresectable metastases, increasing interest hasbeen focused on ablative approaches such as radiofrequency (“RF”),cryotherapy, microwave, or chemical ablation. While interstitialablation can be performed percutaneously or during open surgery,laparoscopic ablation has significant advantages. First, unlikepercutaneous therapy, laparoscopy can identify both hepatic andextrahepatic metastases not visualized on preoperative imaging, whichmisses significant tumors in about 10% to 20% of patients withcolorectal liver metastases. Second, laparoscopic or operativeultrasound (“US”) has been shown to be significantly more accurate thantransabdominal US, CT or MR at visualizing liver lesions. Further,operative approaches, including laparoscopy, permit mobilization ofstructures away from a surface tumor that may be thermally injuredduring RF ablation. Percutaneous ablation and laparoscopic ablation bothtypically require general anesthesia and an overnight hospital stay.Laparoscopy, on the other hand, does not impose a significantly greaterburden on the patient.

While ablation promises advantages compared to other approaches, thetechnical difficulty of manipulating the ultrasound probe, aligning theultrasound probe with the ablation probe, and placement of the ablationprobe demands considerable expertise. The surgeon must precisely placethe ablation probe tip within the volumetric center of the tumor inorder to achieve adequate destruction of the tumor and a 1 cm zone ofsurrounding normal parenchyma. Tumors are identified by preoperativeimaging, primarily CT and MR, and then laparoscopically localized byLUS.

One major limitation of ablative approaches is the lack of accuracy inprobe tip placement within the center of the tumor. This is particularlyimportant, as histologic margins cannot be assessed after ablation as isdone with hepatic resection. In addition, manual guidance often requiresmultiple passes and repositioning of the probe tip, further increasingthe risk of bleeding and tumor dissemination. Intraoperative ultrasoundprovides excellent visualization of tumors and provides guidance for RFprobe placement, but its 2D-nature and dependence on the sonographer'sskill limit its effectiveness.

Although laparoscopic instrumentation and techniques are beginning to beextended to resection of the liver, loss of the surgeon's tactile sensemakes it difficult to assess the safe margins of resection necessary forsafe parenchymal transection. Lack of clear visualization and mapping ofintrahepatic structures with current LUS techniques could result incatastrophic injury to major adjacent structures. The surgeon mustcarefully examine the liver by ultrasound prior to resection in order torule out additional tumors which may preclude curative therapy. Surgeonsalso require ultrasound to determine and plan safe and completeresection with sufficient surgical margin clearance.

Despite its theoretical advantages, intraoperative LUS is not widelypracticed for such uses as laparoscopic liver cancer surgery. To expandusage in this and other applications, advances in LUS robotic surgicalsystems that improve surgeon efficiency in performing minimally invasivesurgical procedures, as well as the ease of using those systems isdesirable.

For example, optimization of LUS for hepatic surgery may significantlyimprove the clinical management of patients. In addition to minimizingmorbidity and discomfort, an improved LUS robotic surgical system maysignificantly reduce costs. Faster, more accurate, and more completeassessment of the liver may be performed by experts, as well aspotentially by surgeons who are not experts in intraoperativeultrasonography of the liver.

Image-guided biopsy of sometimes small and inaccessible liver lesionsmay be facilitated. Advanced LUS robotic tools could increase the use ofresection as a definitive treatment for larger and less favorably placedtumors. Improved real-time guidance for planning, delivery andmonitoring of ablative therapy may also provide the missing tool neededto allow accurate and effective application of this promising therapy.

OBJECTS AND SUMMARY OF THE INVENTION

Accordingly, one object of various aspects of the present invention is alaparoscopic ultrasound robotic surgical system and robotic assistedlaparoscopic ultrasound methods that are easy to use and promote surgeonefficiency.

Another object of various aspects of the present invention is alaparoscopic ultrasound robotic surgical system and robotic assistedlaparoscopic ultrasound methods that provide faster, more accurate andcomplete assessment of anatomic structures.

Another object of various aspects of the present invention is alaparoscopic ultrasound robotic surgical system and robotic assistedlaparoscopic ultrasound methods that provide robotically generatedintra-operative 3D ultrasound images of an anatomic structure usingsurgeon trained trajectories.

Another object of various aspects of the present invention is alaparoscopic ultrasound robotic surgical system and robotic assistedlaparoscopic ultrasound methods that provide flexible display ofultrasound images on a display screen.

Still another object of various aspects of the present invention is alaparoscopic ultrasound robotic surgical system and robotic assistedlaparoscopic ultrasound methods that provide assistance in guiding atool to a target on an anatomic structure.

These and additional objects are accomplished by the various aspects ofthe present invention, wherein briefly stated, one aspect islaparoscopic ultrasound robotic surgical system comprising: a firstrobotic arm mechanically coupled to an ultrasound probe; a secondrobotic arm mechanically coupled to a surgery related device; a mastermanipulator; a control switch having user selectable first and secondmodes; and a processor configured to cause the second robotic arm to belocked in position and the first robotic arm to move the ultrasoundprobe according to user manipulation of the master manipulator when thecontrol switch is in the first mode, and cause the second robotic arm tomanipulate the tool according to manipulation of the master manipulatorand the first robotic arm to move the ultrasound probe according tostored instructions upon detection of a user command associated with thestored instructions when the control switch is in the second mode.

Another aspect is a method for providing robotic assisted laparoscopicultrasound, comprising: storing a current ultrasound probe position andorientation upon detection of a start of training indication; andperiodically storing ultrasound probe positions and orientations todefine a trajectory of positions and orientations until detection of anend of training indication.

Another aspect is a method for providing robotic assisted laparoscopicultrasound, comprising: capturing an ultrasound image using anultrasound probe disposed at a position and orientation; storinginformation of the position and orientation; generating a clickablethumbnail of the ultrasound image; associating the stored position andorientation with the clickable thumbnail; and displaying the clickablethumbnail on a display screen.

Still another aspect is a method for providing robotic assistedlaparoscopic ultrasound, comprising: displaying an ultrasound view of ananatomic structure in a patient as a registered overlay to a camera viewof the anatomic structure; receiving information of a target marked onthe ultrasound view; determining a path for a tool to travel to thetarget within the patient; and generating a virtual fixture to assist inelectronically constraining the tool to travel over the determined path.

Additional objects, features and advantages of the various aspects ofthe present invention will become apparent from the followingdescription of its preferred embodiment, which description should betaken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a top view of an operating room employing alaparoscopic ultrasound robotic surgical system utilizing aspects of thepresent invention.

FIG. 2 illustrates a block diagram of a laparoscopic ultrasound roboticsurgical system utilizing aspects of the present invention.

FIG. 3 illustrates a laparoscopic ultrasound probe utilizing aspects ofthe present invention.

FIG. 4 illustrates a flow diagram of a method for training a LUS roboticsurgical system to robotically move a LUS probe in a trained manner uponcommand, utilizing aspects of the present invention.

FIG. 5 illustrates a flow diagram of a method for generating a clickablethumbnail image that allows a user to command that a LUS probe beautomatically moved to a position and orientation from which the imagewas captured, utilizing aspects of the present invention.

FIG. 6 illustrates a flow diagram of a method for automatically moving sLUS probe to a position and orientation associated with a clickablethumbnail image, utilizing aspects of the present invention.

FIG. 7 illustrates a flow diagram of a method for robotically assistedneedle guidance to a marked lesion of a cancerous structure, utilizingaspects of the present invention.

FIG. 8 illustrates a perspective view of a 3D ultrasound image of ananatomic structure in a camera reference frame with selectable 2D imageslices as used in a medical robotic system utilizing aspects of thepresent invention.

FIG. 9 illustrates a perspective view of a 3D camera view of an anatomicstructure in a camera reference as used in a medical robotic systemutilizing aspects of the present invention.

FIG. 10 illustrates a perspective view of a frontal 2D slice of a 3Dultrasound view of an anatomic structure that overlays a 3D camera viewof the anatomic structure, as displayable in a medical robotic systemutilizing aspects of the present invention.

FIG. 11 illustrates a perspective view of an inner 2D slice of a 3Dultrasound view of an anatomic structure that overlays a 3D camera viewof the anatomic structure, as displayable in a medical robotic systemutilizing aspects of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 illustrates, as an example, a top view of an operating roomemploying a robotic surgical system. The robotic surgical system in thiscase is a Laparascopic Ultrasound Robotic Surgical System 100 includinga Console (“C”) utilized by a Surgeon (“S”) while performing a minimallyinvasive diagnostic or surgical procedure with assistance from one ormore Assistants (“A”) on a Patient (“P”) who is reclining on anOperating table (“O”).

The Console includes a Master Display 104 (also referred to herein as a“Display Screen”) for displaying one or more images of a surgical sitewithin the Patient as well as perhaps other information to the Surgeon.Also included are Master Input Devices 107 and 108 (also referred toherein as “Master Manipulators”), one or more Foot Pedals 105 and 106, aMicrophone 103 for receiving voice commands from the Surgeon, and aProcessor 102. The Master Input Devices 107 and 108 may include any oneor more of a variety of input devices such as joysticks, gloves,trigger-guns, hand-operated controllers, or the like. The Processor 102is preferably a personal computer that may be integrated into theConsole or otherwise connected to it in a conventional manner.

The Surgeon performs a minimally invasive surgical procedure bymanipulating the Master Input Devices 107 and 108 so that the Processor102 causes their respectively associated Slave Arms 128 and 129 (alsoreferred to herein as “Slave Manipulators”) to manipulate theirrespective removably coupled and held Surgical Instruments 138 and 139(also referred to herein as “Tools”) accordingly, while the Surgeonviews three-dimensional (“3D”) images of the surgical site on the MasterDisplay 104.

The Tools 138 and 139 are preferably Intuitive Surgical's proprietaryEndoWrist™ articulating instruments, which are modeled after the humanwrist so that when added to the motions of the robot arm holding thetool, they allow a full six degrees of freedom of motion, which iscomparable to the natural motions of open surgery. Additional details onsuch tools may be found in commonly owned U.S. Pat. No. 5,797,900entitled “Wrist Mechanism for Surgical Instrument for PerformingMinimally Invasive Surgery with Enhanced Dexterity and Sensitivity,”which is incorporated herein by this reference. At the operating end ofeach of the Tools 138 and 139 is a manipulatable end effector such as aclamp, grasper, scissor, stapler, blade, needle, or needle holder.

The Master Display 104 has a high-resolution stereoscopic video displaywith two progressive scan cathode ray tubes (“CRTs”). The system offershigher fidelity than polarization, shutter eyeglass, or othertechniques. Each eye views a separate CRT presenting the left or righteye perspective, through an objective lens and a series of mirrors. TheSurgeon sits comfortably and looks into this display throughout surgery,making it an ideal place for the Surgeon to display and manipulate 3-Dintraoperative imagery.

A Stereoscopic Endoscope 140 (also referred to as a “Laparoscope”)provides right and left camera views to the Processor 102 so that it mayprocess the information according to programmed instructions and causeit to be displayed on the Master Display 104. A Laparoscopic Ultrasound(“LUS”) Probe 150 provides two-dimensional (“2D”) ultrasound imageslices of an anatomic structure to the Processor 102 so that theProcessor 102 may generate a 3D ultrasound computer model of theanatomic structure and cause the 3D computer model (or alternatively, 2D“cuts” of it) to be displayed on the Master Display 104 as an overlay tothe endoscope derived 3D images or within a Picture-in-Picture (“PIP”)in either 2D or 3D and from various angles and/or perspectives accordingto Surgeon or stored program instructions.

Each of the Tools 138 and 139, as well as the Endoscope 140 and LUSProbe 150, is preferably inserted through a cannula or trocar (notshown) or other tool guide into the Patient so as to extend down to thesurgical site through a corresponding minimally invasive incision suchas Incision 166. Each of the Slave Arms 121-124 is conventionally formedof linkages which are coupled together and manipulated through motorcontrolled joints (also referred to as “active joints”). Setup Arms (notshown) comprising linkages and setup joints are used to position theSlave Arms 121-124 vertically and horizontally so that their respectivesurgical related instruments may be coupled for insertion into thecannulae.

The number of surgical tools used at one time and consequently, thenumber of slave arms being used in the System 100 will generally dependon the diagnostic or surgical procedure and the space constraints withinthe operating room, among other factors. If it is necessary to changeone or more of the tools being used during a procedure, the Assistantmay remove the tool no longer being used from its slave arm, and replaceit with another tool, such as Tool 131, from a Tray (“T”) in theOperating Room.

Preferably, the Master Display 104 is positioned near the Surgeon'shands so that it will display a projected image that is oriented so thatthe Surgeon feels that he or she is actually looking directly down ontothe surgical site. To that end, an image of the Tools 138 and 139preferably appear to be located substantially where the Surgeon's handsare located even though the observation points (i.e., that of theEndoscope 140 and LUS Probe 150) may not be from the point of view ofthe image.

In addition, the real-time image is preferably projected into aperspective image such that the Surgeon can manipulate the end effectorof a Tool, 138 or 139, through its associated Master Input Device, 107or 108, as if viewing the workspace in substantially true presence. Bytrue presence, it is meant that the presentation of an image is a trueperspective image simulating the viewpoint of an operator that isphysically manipulating the Tools. Thus, the Processor 102 transformsthe coordinates of the Tools to a perceived position so that theperspective image is the image that one would see if the Endoscope 140was looking directly at the Tools from a Surgeon's eye-level during anopen cavity procedure.

The Processor 102 performs various functions in the System 100. Oneimportant function that it performs is to translate and transfer themechanical motion of Master Input Devices 107 and 108 to theirassociated Slave Arms 121 and 122 through control signals over Bus 110so that the Surgeon can effectively manipulate their respective Tools138 and 139. Another important function is to implement the variousmethods described herein providing a robotic assisted LUS capability.

Although described as a processor, it is to be appreciated that theProcessor 102 may be implemented in practice by any combination ofhardware, software and firmware. Also, its functions as described hereinmay be performed by one unit, or divided up among different components,each of which may be implemented in turn by any combination of hardware,software and firmware.

Prior to performing a minimally invasive surgical procedure, ultrasoundimages captured by the LUS Probe 150, right and left 2D camera imagescaptured by the stereoscopic Endoscope 140, and end effector positionsand orientations as determined using kinematics of the Slave Arms121-124 and their sensed joint positions, are calibrated and registeredwith each other.

In order to associate the ultrasound image with the rest of the surgicalenvironment, both need to be expressed in the same coordinate frame.Typically, the LUS Probe 150 is either labeled with markers and trackedby a tracking device such as the Optotrak® position sensing systemmanufactured by Northern Digital Inc. of Ontario, Canada, or held by arobot with precise joint encoders. Then the rigid transformation betweenthe ultrasound image and the frame being tracked is determined (which istypically referred to as the ultrasound calibration).

For example, using the Optotrak® frame for the ultrasound calibration,the ultrasound image generated by the LUS Probe 150 is calibrated to anOptotrak® rigid body using an AX=XB formulation. “AX=XB” is a rubric fora class of calibration/registration problem commonly encountered incomputer vision, surgical navigation, medical imaging, and robotics. Themathematical techniques are well known. See, e.g., E. Boctor, A.Viswanathan, M. Chioti, R. Taylor, G. Fichtinger, and G. Hager, “A NovelClosed Form Solution for Ultrasound Calibration,” InternationalSymposium on Biomedical Imaging, Arlington, Va., 2004, pp. 527-530.

“A” and “B” in this case, are transformations between poses of theOptotrak® rigid body (A) and the ultrasound image (B). Thus, “X” is thetransformation from the ultrasound image to the rigid body.

To perform the ultrasound calibration, the LUS Probe 150 may be placedin three known orientations defined by the AX=XB calibration phantom.The ultrasound image frame may then be defined by three fiducials whichappear in each of the three poses. The three poses allow three relativetransformations based on Optotrak® readings (A) and three relativetransformations based on the ultrasound images (B) for the AX=XBregistration.

Camera calibration is a common procedure in computer visionapplications. As an example, in order to determine the intrinsic andextrinsic parameters of the Endoscope 140, a checkerboard phantom with amulti-plane formulation provided by the Caltech Camera CalibrationToolbox may be used. To construct the phantom, Optotrak® markers areadded to a typical checkerboard video calibration phantom, and eachcorner of the checkerboard is digitized using a calibrated Optotrak®pointer. Thus, the corner positions may be reported with respect to theOptotrak®.

The calibration may then be performed by placing the phantom in view ofthe Endoscope 140 in several dozen orientations, and recording bothstereo image data and Optotrak® readings of the four checkerboardcorners. The images may then be fed into the calibration toolbox, whichdetermines the intrinsic and extrinsic camera parameters, as well as the3D coordinates of the grid corners in the camera frame. Thesecoordinates may then be used with the Optotrak® readings to perform apoint-cloud to point-cloud registration between the Endoscope 140 rigidbody and camera frame.

The Controller 102 is configured to use the robot kinematics to report acoordinate frame for the LUS Probe 150 tip relative to the Endoscope140. However, due to inaccuracies in the setup joint encoders, both ofthese coordinate frames may be offset from their correct values. Thus,it may be necessary to register the offsets between the real cameraframe of the Endoscope 140 and the camera frame calculated from thekinematics as well as between the real and kinematic LUS Probe 150frames. With this complete, the kinematics may be used in place of theOptotrak® readings to determine ultrasound image overlay placement.

As long as the position of the Endoscope 140 doesn't overly change, aconstant transformation may be assumed between the kinematic tool tipand the laparoscopic Optotrak® rigid body. Using an AX=XB formulation,the LUS Probe 150 may be moved, for example, to several positions, andthe static offset between the tool tip and Optotrak® rigid bodyregistered. Knowing this offset, the Endoscope 140 offset may becalculated directly:C _(CD) =D _(LusD)(C _(LusUrb))⁻¹ T _(Ourb)(T _(OErb))⁻¹ F _(CErb)  (1)

where C_(CD) is the camera offset from the real Endoscope 140 (alsoreferred to herein simply as the “camera”) frame to the camera framecalculated from the kinematics, F_(CErb) is the transformation from thecamera to the endoscope rigid body, T_(Ourb)·(T_(OErb))⁻¹ is thetransformation from the camera rigid body to the LUS rigid body,C_(LusUrb) is the transformation from the LUS rigid body to thekinematic ultrasound tool tip, and D_(LusD) is the reading from theController 102 giving the transformation from the kinematic ultrasoundtool tip to a fixed reference point associated with the Slave Arms121-124.

However, the aforedescribed registration should be redone each time thecamera is moved, thus making it best suited for pre-operativecalibration and registration. For intra-operative, the registration maybe better performed using video tracking of a visual marker on the LUSProbe 150 instead of the Optotrak® readings. Thus, if the camera weremoved while using tool tracking, the registration can be corrected onthe fly as the tool is tracked. For additional details on tool tracking,see, e.g., commonly owned U.S. patent application Ser. No. 11/130,471entitled “Methods and system for performing 3-D tool tracking by fusionof sensor and/or camera derived data during minimally invasive surgery,”filed May 16, 2005, which is incorporated herein by reference. Inaddition to, or alternatively, manual registration of ultrasound andcamera images may be performed using conventional grab, move and rotateactions on a 3D ultrasound computer model of an anatomic structure, sothat the computer model is properly registered over a camera model ofthe anatomic structure in the Master Display 104.

Slave Arms 123 and 124 may manipulate the Endoscope 140 and LUS Probe150 in similar manners as Slave Arms 121 and 122 manipulate Tools 138and 139. When there are only two master input devices in the system,however, such as Master Input Devices 107 and 108 in the System 100, inorder for the Surgeon to manually control movement of either theEndoscope 140 or LUS Probe 150, it may be required to temporarilyassociate one of the Master Input Devices 107 and 108 with the Endoscope140 or the LUS Probe 150 that the Surgeon desires manual control over,while its previously associated Tool and Slave Manipulator are locked inposition.

FIG. 2 illustrates, as an example, a block diagram of the LUS RoboticSurgical System 100. In this system, there are two Master Input Devices107 and 108. Master Input Device 107 controls movement of either a Tool138 or a stereoscopic Endoscope 140, depending upon which mode itsControl Switch Mechanism 211 is in, and Master Input Device 108 controlsmovement of either a Tool 139 or a LUS Probe 150, depending upon whichmode its Control Switch Mechanism 231 is in.

The Control Switch Mechanisms 211 and 231 may be placed in either afirst or second mode by a Surgeon using voice commands, switchesphysically placed on or near the Master Input Devices 107 and 108, FootPedals 105 and 106 on the Console, or Surgeon selection of appropriateicons or other graphical user interface selection means displayed on theMaster Display 104 or an auxiliary display (not shown).

When Control Switch Mechanism 211 is placed in the first mode, it causesMaster Controller 202 to communicate with Slave Controller 203 so thatmanipulation of the Master Input 107 by the Surgeon results incorresponding movement of Tool 138 by Slave Arm 121, while the Endoscope140 is locked in position. On the other hand, when Control SwitchMechanism 211 is placed in the second mode, it causes Master Controller202 to communicate with Slave Controller 233 so that manipulation of theMaster Input 107 by the Surgeon results in corresponding movement ofEndoscope 140 by Slave Arm 123, while the Tool 138 is locked inposition.

Similarly, when Control Switch Mechanism 231 is placed in the firstmode, it causes Master Controller 222 to communicate with SlaveController 223 so that manipulation of the Master Input 108 by theSurgeon results in corresponding movement of Tool 139 by Slave Arm 122.In this case, however, the LUS Probe 150 is not necessarily locked inposition. Its movement may be guided by an Auxiliary Controller 242according to stored instructions in Memory 240. The Auxiliary Controller242 also provides haptic feedback to the Surgeon through Master Input108 that reflects readings of a LUS Probe Force Sensor 247. On the otherhand, when Control Switch Mechanism 231 is placed in the second mode, itcauses Master Controller 222 to communicate with Slave Controller 243 sothat manipulation of the Master Input 222 by the Surgeon results incorresponding movement of LUS Probe 150 by Slave Arm 124, while the Tool139 is locked in position.

Before switching back to the first or normal mode, the Master InputDevice 107 or 108 is preferably repositioned to where it was before theswitch to the second mode of Control Switch 211 or 231, as the case maybe, or kinematic relationships between the Master Input Device 107 or108 and its respective Tool Slave Arm 121 or 122 is readjusted so thatupon switching back to the first or normal mode, abrupt movement of theTool 138 or 139 does not occur. For additional details on controlswitching, see, e.g., commonly owned U.S. Pat. No. 6,659,939“Cooperative Minimally Invasive Telesurgical System,” which isincorporated herein by this reference.

The Auxiliary Controller 242 also performs other functions related tothe LUS Probe 150 and the Endoscope 140. It receives output from a LUSProbe Force Sensor 247, which senses forces being exerted against theLUS Probe 150, and feeds the force information back to the Master InputDevice 108 through the Master Controller 222 so that the Surgeon mayfeel those forces even if he or she is not directly controlling movementof the LUS Probe 150 at the time. Thus, potential injury to the Patientis minimized since the Surgeon has the capability to immediately stopany movement of the LUS Probe 150 as well as the capability to take overmanual control of its movement.

Another key function of the Auxiliary Control 242 is to cause processedinformation from the Endoscope 140 and the LUS Probe 150 to be displayedon the Master Display 104 according to user selected display options. Aswill be described in more detail below, such processing includesgenerating a 3D ultrasound image from 2D ultrasound image slicesreceived from the LUS Probe 150 through an Ultrasound Processor 246,causing either 3D or 2D ultrasound images corresponding to a selectedposition and orientation to be displayed in a picture-in-picture windowof the Master Display 104, and causing either 3D or 2D ultrasound imagesof an anatomic structure to overlay a camera captured image of theanatomic structure being displayed on the Master Display 104.

Although shown as separate entities, the Master Controllers 202 and 222,Slave Controllers 203, 233, 223, and 243, and Auxiliary Controller 242are preferably implemented as software modules executed by the Processor102, as well as certain mode switching aspects of the Control SwitchMechanisms 211 and 231. The Ultrasound Processor 246 and Video Processor236, on the other hand, are separate boards or cards typically providedby the manufacturers of the LUS Probe 150 and Endoscope 140 that areinserted into appropriate slots coupled to or otherwise integrated withthe Processor 102 to convert signals received from these image capturingdevices into signals suitable for display on the Master Display 104and/or for additional processing by the Auxiliary Controller 242 beforebeing displayed on the Master Display 104.

FIG. 3 illustrates a side view of one embodiment of the LUS Probe 150.The LUS Probe 150 is a dexterous tool with preferably two distal degreesof freedom, permitting reorientation of LUS Sensor 301 through, forexample, approximately ±80° in distal “pitch” and “yaw”, and ±240° in“roll” about a ball joint type, pitch-yaw mechanism 311 (functioning asand also referred to herein as a “Wrist” mechanism). Opposing pairs ofDrive Rods or Cables (not shown) physically connected to a proximal endof the LUS Sensor 301 and extending through an internal passage ofElongated Shaft 312 mechanically control pitch and yaw movement of theLUS Sensor 301 using conventional push-pull type action. Thisflexibility of the LUS Probe 150 (provided by the pitch/yaw wristmechanism) is especially useful in optimally orienting the LUS Probe 150for performing ultrasonography on an anatomic structure during aminimally invasive surgical procedure.

The LUS Sensor 301 captures 2D ultrasound slices of a proximate anatomicstructure, and transmits the information back to the Processor 102through LUS Cable 304. Although shown as running outside of theElongated Shaft 312, the LUS Cable 304 may also extend within it. AClamshell Sheath 321 encloses the Elongate Shaft 312 and LUS Cable 304to provide a good seal passing through a Cannula 331 (or trocar).Fiducial Marks 302 and 322 are placed on the LUS Sensor 301 and theSheath 321 for video tracking purposes.

A force sensing capability is provided by Strain Gauges 303 whichprovide direct feedback of how hard the LUS Probe 150 is pushing on astructure being sonographed, supplementing whatever limited feedback isavailable from joint motor torques. Potential uses of this informationinclude: providing a redundant safety threshold check warning theSurgeon or preventing motion into the structure if forces get too great;providing the Surgeon with an approved haptic appreciation of how hardhe or she is pushing on a structure; and possibly permitting somemeasure of compensation for unmodeled deflections of the Pitch-Yaw or“Wrist” Mechanism 311 which are not detected for some reason by jointposition sensors or encoders. The Strain Gauges 303 in this case servethe function of the LUS Probe Force Sensor 247 as previously describedin reference to FIG. 2.

Robotic assisted LUS has the potential to reduce variability in theultrasound images produced, compared to freehand scanning, and canreduce operator workload and difficulty. Behaviors as simple as rockingthe LUS Probe 150 back and forth can maintain an updated 3D ultrasoundimage without operator intervention. More complicated behaviors caninclude movement of the LUS Probe 150 along the surface of a targetanatomical structure in a methodical pattern to generate a full image ofthe target, or reliably returning to a previously scanned probe locationand orientation.

FIG. 4 illustrates, as an example, a flow diagram of a method fortraining the Auxiliary Controller 242 (i.e., providing it with storedinstructions) to cause the LUS Probe 150 to be robotically moved in thetrained manner upon command, in order to capture a sequence of 2Dultrasound image slices of an anatomic structure, which are used by theAuxiliary Controller 242 to generate a 3D computer model of thestructure. Prior to performing the training, the Control SwitchMechanism 231 is placed in its second mode so that the Surgeon may movethe LUS Probe 150 for training purposes by manipulating the Master InputDevice 108. After performing training, the Control Switch Mechanism 231is then placed back into its first or normal mode so that the Surgeonmay manipulate the Tool 139 to perform a minimally invasive surgicalprocedure using the Master Input Device 108.

In 401, the training module is initially idle (i.e., it is not beingexecuted by the Processor 102). In 402, the Processor 102 (or a trainingmodule agent running in the background) may periodically check whether astart of training indication is received. Alternatively, the start oftraining indication may act as an interrupt which initiates running ofthe training module. The start of training indication may be initiatedby a Surgeon through a recognized voice command, selection of a trainingoption on a graphical user interface displayed on the Master Display104, a switch mechanism that may physically be located on thecorresponding Master Control Input 108 or other convenient locationaccessible to the Surgeon, or any other conventional means.

After the start of training indication is detected, in 403, the trainingmodule records or stores the current LUS Probe 150 position andorientation, and periodically (or upon Surgeon command) continues to doso by looping around 403 and 404 until a stop training indication isdetected or received. The stop training indication in this case may alsobe initiated by the Surgeon in the same manner as the start of trainingindication, or it may be initiated in a different, but otherconventional manner. After the stop training indication is detected orreceived, a last position and orientation of the LUS Probe 150 isrecorded or stored.

Between the start and stop of training, the Surgeon moves the LUS Probe150 and the Processor 102 stores its trajectory of points andorientations so that they may be retraced later upon command. In onetype of training, the Surgeon moves the LUS Probe 150 back and forthnear an anatomic structure in order to capture a sequence of 2Dultrasound image slices from which a 3D version (or computer model) ofthe anatomic structure may be rendered by the Processor 102. In anothertype of training, the Surgeon move the LUS Probe 150 once or more timesalong the surface of the anatomic structure in order to capture adifferent sequence of 2D ultrasound image slices from which a 3D version(or computer model) of the anatomic structure may be rendered by theProcessor 102.

Although described as recording the positions and orientations of theLUS Probe 150, in practice, the active joint positions of its Slave Arm124 are stored instead since their measurements are directly obtainablethrough encoders attached to each of the joints and their positionscorrespond to the LUS Probe 150 positions and orientations.

After storing the trajectory of positions and orientations of the LUSProbe 150 in the Memory 240, the trajectory is then associated with ameans for the Surgeon to command the Auxiliary Controller 242 to movethe LUS Probe 150 in the desired fashion. For example, the trajectorymay be associated with a voice command which upon its detection, theAuxiliary Controller 242 causes the Slave Arm 124 to move the LUS Probe150 back and forth along the stored trajectory of positions andorientations. Likewise, the trajectory may also be associated with auser selectable option on a graphical user interface displayed on theMaster Display 104, or it may be associated with a switch mechanism suchas a button or unused control element on the Master Input Device 108. Itmay also be associated with the depression of the Foot Pedal 106, sothat the Auxiliary Controller 242 causes the Slave Arm 124 to move theLUS Probe 150 back and forth along the stored trajectory of positionsand orientations as long as the Foot Pedal 106 is being depressed, andstops such motion once the Surgeon takes his or her foot off the FootPedal 106.

FIG. 5 illustrates, as an example, a flow diagram of a method forgenerating clickable thumbnail images corresponding to LUS Probe 150positions and orientations that are stored in Memory 240, so that whenthe Surgeon clicks on one of the thumbnail images, the AuxiliaryController 242 causes the Slave Arm 124 to move the LUS Probe 150 to itsstored position and orientation. This allows the Surgeon to move the LUSProbe 150 to see different views of an anatomic structure while theControl Switch Mechanism 231 is in its first or normal mode. Thus, theSurgeon can continue to perform a minimally invasive surgical procedureby manipulating Tool 139 using the Master Input Device 108. The methodmay then be combined with that described in reference to FIG. 4 in orderto generate a sequence of 2D ultrasound image slices starting from thatposition and orientation, from which the Auxiliary Controller 242 maygenerate a 3D computer model rendition of the anatomic structure.

Prior to performing the method, however, the Control Switch Mechanism231 is placed in its second mode so that the Surgeon may move the LUSProbe 150 into the desired positions and orientations by manipulatingthe Master Input Device 108. After generating the clickable thumbnailimages, the Control Switch Mechanism 231 is then placed back into itsfirst or normal mode so that the Surgeon may manipulate the Tool 139 toperform the minimally invasive surgical procedure using the Master InputDevice 108.

In 501, the Auxiliary Controller 242 receives a snapshot command fromthe Surgeon. The snapshot command may be, for example, a voice command,graphical user interface selection, or switch position. In 502, theAuxiliary Controller 242 causes the LUS Probe 150 to capture a 2Dultrasound image slice, and in 503, a thumbnail of the image isgenerated. The thumbnail in this case may include a simple JPEG or GIFfile of the captured image. In 504, the current position and orientationof the LUS Probe 150 is stored in Memory 240 along with information ofits association with the thumbnail. In 505, a clickable version of thethumbnail is displayed on the Master Display 104, so that the Surgeonmay command the Auxiliary Controller 242 to cause the LUS Probe to bepositioned and oriented at the stored position and orientation at anytime upon clicking with his or her mouse or other pointing device on theclickable thumbnail. The Surgeon may then move the LUS Probe 150 toother positions and/or orientations, and repeat 501-505 to generateadditional thumbnail images.

FIG. 6 illustrates, as an example, a flow diagram of a method forautomatically moving the LUS Probe 150 to a position and orientationassociated with a clickable thumbnail upon command to do so by a Surgeonwhile performing a minimally invasive surgical procedure using Tool 139.In 601, the clicking of a thumbnail generated by the method described inreference to FIG. 5 is detected by, for example, a conventionalinterrupt handling process. Upon such detection, in 602, the AuxiliaryController 242 is instructed by, for example, stored instructionscorresponding to the interrupt handling process, to retrieve theposition and orientation stored in Memory 240 which is associated withthe thumbnail. The Auxiliary Controller 242 then causes the LUS Probe150 to move to that position and orientation by appropriatelycontrolling Slave Arm 124. Thus, the Surgeon is able to move the LUSProbe 150 to a desired position without having to change modes of theControl Switch Mechanism 231 and halt operation of the Tool 139 untilthe LUS Probe 150 is moved.

FIG. 7 illustrates, as an example, a flow diagram of a method forrobotically assisted needle guidance and penetration into a markedlesion of a cancerous structure, which allows appreciation for theaspects of robotic assisted LUS described herein. In 701, a selected 2Dultrasound image slice view of a cancerous structure such as a liver isdisplayed at the proper depth on the Master Display 104 as an overlay toa 3D camera view of the cancerous structure. The selected 2D ultrasoundimage slice view may be a frontal view or an inner slice view as takenfrom a previously generated 3D ultrasound computer model of thecancerous structure.

As an example clarifying the 701 process, FIG. 8 illustrates asimplified perspective view of a 3D ultrasound computer model 800 of thecancerous structure, which has been generated, for example, using themethod described in reference to FIG. 4, and has been translated intothe camera reference frame (EX, EY, EZ). FIG. 9, on the other hand,illustrates a simplified perspective view of a 3D camera view 900 of thecancerous structure as taken by the stereoscopic Endoscope 140. If theSurgeon selects a frontal slice 801 of the 3D ultrasound computer model800 to be viewed as an overlay to the 3D camera view 900, then theoverlay will appear as shown in FIG. 10. On the other hand, if theSurgeon selects one of the inner slices 802-804 of the 3D ultrasoundcomputer model 800, such as inner slice 803, to be viewed as an overlayto the 3D camera view 900, then the overlay will appear as shown in FIG.11 with the 2D ultrasound image slice 803 displayed at the proper depth.To avoid confusion, the portion of the 3D camera view above that depthis made transparent.

Alternatively, in 701, the surgeon may manually control movement of theLUS Probe 150 so that 2D ultrasound image slices captured by it appearas emanating in proper perspective and direction from the 3D cameraimage of the LUS Probe 150 in the Master Display 104. Preferably, theemanated 2D image slices being displayed in the Master Display 104 donot occlude the anatomic structure being probed. This manual approachmay be particularly useful to the Surgeon for quickly spotting lesionsin the anatomic structure.

In 702, the Surgeon marks lesions on the cancerous structure displayedas a result of 701. Each marked lesion is preferably marked using adesignated color in order to clearly show that the Surgeon has alreadyidentified it, thereby avoiding double counting. The location in thecamera reference frame (EX, EY, EZ) of each marked lesion is stored inMemory 240, and in 703, the Processor 102 determines an optimal needletip path to that location.

In 703, the Processor 102 generates a virtual fixture to help guide theneedle to the marked lesion. To generate the virtual fixture, localkinematic constraints on the Slave Arm manipulating the needle Tool maybe specified by providing a table of constraints of the form:({right arrow over (x)}−{right arrow over (x)} ₀)^(T) A _(K)({rightarrow over (x)}−{right arrow over (x)} ₀)+{right arrow over (b)}_(K)({right arrow over (x)}−{right arrow over (x)} ₀)≤c  (2)

where {right arrow over (x)} represents, in simplified terms, thecurrent 6 DOF kinematic pose of a master arm, or, in more general terms,a parameterization of a Cartesian pose F linearized about some nominalpose F₀ so that ({right arrow over (x)}−{right arrow over (x)}₀)˜F₀ ⁻¹F.The tables are to be updated periodically based on visual feedback, userinteraction, etc.

As can be appreciated, equation (2) can be easily checked and enforced.

Similarly, a simple table-driven interface for surgeon interactionforces can be implemented approximately as follows:

{right arrow over (f)} ← 0; y ← {right arrow over (x)} − {right arrowover (x)}₀ ; (3) for k ← 1 to N do { ε ← {right arrow over (y)}^(T)C_(K) {right arrow over (y)} + {right arrow over (d)}_(K) {right arrowover (y)} − e_(K) ; if ε > 0 then {{right arrow over (g)} ← 2 C_(K){right arrow over (y)} {right arrow over (d)}_(K) ; {right arrow over(f)} ← {right arrow over (f)} + f (ε){right arrow over (g)}/∥ {rightarrow over (g)} ∥; };  }; output {right arrow over (f)} (after limiting& spacing )

where ε corresponds, roughly, to a distance from a surface in statespace and the function ƒ(ε) corresponds to a (non-linear) stiffness.

The above formulation suffices to support a variety of virtual chamfers,virtual springs, detents, etc. It is also easily extended to virtualdampers by adding velocity terms.

Now, more particularly, in the present case where it is desired to helpaim an injection needle at a target in a live ultrasound image, let:

$\begin{matrix}\begin{matrix}{{\overset{->}{P}}_{TROCAR} = {{position}\mspace{14mu}{where}\mspace{14mu}{needle}\mspace{14mu}{enters}\mspace{14mu}{patient}}} \\{= {{``{RCM}"}\mspace{14mu}{point}\mspace{14mu}{for}\mspace{14mu}{needle}\mspace{14mu}{insertion}\mspace{14mu}{arm}}}\end{matrix} & (4) \\{R_{NEEDLE} = {{R_{0}{R( \overset{->}{\alpha} )}} = {{orientation}\mspace{14mu}{of}\mspace{14mu}{needle}\mspace{14mu}{arm}}}} & (5) \\{\overset{->}{\alpha} = {{vector}\mspace{14mu}{representation}\mspace{14mu}{for}\mspace{14mu}{small}\mspace{14mu}{rotation}}} & (6) \\{F_{LUS} = {\lbrack {R_{LUS},{\overset{->}{P}}_{LUS}} \rbrack = {{pose}\mspace{14mu}{of}\mspace{14mu}{LUS}\mspace{14mu}{sensor}}}} & (7) \\{V_{TARGET} = {{position}\mspace{14mu}{of}\mspace{14mu}{target}\mspace{14mu}{wrt}\mspace{14mu}{LUS}\mspace{14mu}{sensor}}} & (8)\end{matrix}$

Then the basic constraint is that the needle axis (which is assumed forthis example to be the {right arrow over (Z)}axis of the needle driver)should be aimed at the target lesion, which will be given by F_(LUS){right arrow over (v)}_(TARGET). One metric for the aiming directionerror will be:

$\begin{matrix}\begin{matrix}{{ɛ_{AIMING}( \overset{->}{\alpha} )} = {{( {R_{NEEDLE}\overset{->}{z}} ) \times ( {{F_{LUS}{\overset{->}{v}}_{TARGET}} - {\overset{->}{P}}_{TROCAR}} )}}^{2}} \\{= {{( {{R( \overset{->}{\alpha} )}\overset{->}{z}} ) \times {R_{0}^{- 1}( {{F_{LUS}{\overset{->}{v}}_{TARGET}} - {\overset{->}{P}}_{TROCAR}} )}}}^{2}}\end{matrix} & (9)\end{matrix}$

which can be approximated as a quadratic form in {right arrow over (α)}and converted to a virtual fixture using the method described above.Similarly, if the position of the needle tip is {right arrow over(P)}_(TIP), the penetration depth beyond the LUS target will be givenby:ε_(BEYOND)=(R ₀ R({right arrow over (α)}){right arrow over (z)})·(F_(LUS) {right arrow over (v)} _(TARGET) −{right arrow over (P)}_(TIP))  (10)

which can easily be transcribed into a virtual detent or barrierpreventing over-penetration. Alternatively, a simple spherical attractorvirtual fixture can be developed to minimize ∥F_(LUS) v _(TARGET)−{rightarrow over (P)}_(TIP)∥.

In 705, the Processor 102 determines the needle tip position as it movestowards the target lesion, and in 706, the Processor 102 determines thedistance between the needle tip position and the target lesion. Theneedle tip position may be determined from the Slave Arm kinematicsand/or through visual tracking in the camera image.

In 707, the color of the lesion or some other object in the displaychanges as the needle tip gets closer to the target. For example, thecolor may start off as blue when the needle tip is still far away fromthe target, and it may change through color spectrum so that it becomesred as it nears the target. Alternatively, a bar graph or other visualindicator may be used to give a quick sense of the distance.

In 708, a determination is made whether the distance has reached athreshold distance (usually specified as some distance close to or evenat the surface of the target lesion). If the threshold has not beenreached, then the method loops back to 705 and continually repeats705-708 until the threshold is reached. Once the threshold is reached,in 709, a 90 degree view of the cancerous structure and the approachingneedle is shown in a picture-in-picture window of the Master Display104. The method may then go back to 705 and repeat 705-708 as the needlepenetrates the cancerous structure or withdraws back to its startposition.

Although the various aspects of the present invention have beendescribed with respect to a preferred embodiment, it will be understoodthat the invention is entitled to full protection within the full scopeof the appended claims.

What is claimed is:
 1. A method for providing robotic assistedlaparoscopic ultrasound, comprising: displaying an ultrasound view of ananatomic structure in a patient as a registered overlay to a camera viewof the anatomic structure on a display; receiving information of atarget marked on the ultrasound view; determining a path for a tool totravel to the target within the patient; generating a virtual fixture toassist in electronically constraining the tool to travel over thedetermined path; determining a distance of the tool from the target; andchanging a color of the target marked on the ultrasound view so as toindicate the distance on the display.
 2. A method for providing roboticassisted laparoscopic ultrasound, comprising: displaying an ultrasoundview of an anatomic structure in a patient as a registered overlay to acamera view of the anatomic structure on a display; receivinginformation of a target marked on the ultrasound view; determining apath for a tool to travel to the target within the patient; determiningwhen a tip of the tool reaches a threshold distance from a surface ofthe anatomic structure; and automatically displaying a right angle viewof the ultrasound image and the tip of the tool, upon determining thatthe tip of the tool has reached the threshold distance, in apicture-in-picture window on the display.
 3. The method according toclaim 1, wherein the anatomic structure is cancerous, the target is alesion marked by a surgeon on the displayed ultrasound view, and thetool is a needle.
 4. The method according to claim 1, furthercomprising: determining when a tip of the tool reaches a thresholddistance from a surface of the anatomic structure; and automaticallydisplaying a right angle view of the ultrasound image and the tip of thetool, upon determining that the tip of the tool has reached thethreshold distance, in a picture-in-picture window on the display. 5.The method according to claim 1, wherein the ultrasound image is athree-dimensional image generated from a sequence of two-dimensionalultrasound image slices of the anatomic structure.
 6. The methodaccording to claim 2, wherein the anatomic structure is cancerous, thetarget is a lesion marked by a surgeon on the displayed ultrasound view,and the tool is a needle.
 7. The method according to claim 2, furthercomprising: determining a distance of the tool from the target; andchanging a color of the target marked on the ultrasound view on thedisplay so as to indicate the distance.
 8. The method according to claim2, further comprising: generating a virtual fixture to assist inelectronically constraining the tool to travel over the determined path.9. The method according to claim 2, wherein the ultrasound image is athree-dimensional image generated from a sequence of two-dimensionalultrasound image slices of the anatomic structure.